1. Field of the Invention
The present invention relates generally to a low-complexity and high-quality error concealment techniques for video sequence transmissions, and more particularly, to a spatial error concealment (SEC) method for concealing a spatial image error of an encoded image frame caused by damaged macroblocks (MB).
2. The Prior Arts
Image transmission service including video meeting, website browsing, image file transmission, is a very important service provided by communication and IT enterprises. Typically, an original image file desired to be transmitted is often too large to be effectively transmitted over the internet, or often occupies too much memory space. As such, an image file is often performed with a coding process with a high compression ratio and a low loss ratio before transmission for reducing the size of the image content. Such a coding process may be selected from TIFF, JPEG, MJEP, MPEG, H.264/AVC, in accordance with the demands of the system for the image compression ratio, the loss ratio, so as to obtain an optimal algorithm for static images or continuous images.
However, when being transmitted, the encoded image file is often interfered by the transmission interface or the electrical system of the receiving end. Such interferences usually cause damages or errors of the image file content, so that the receiving end becomes incapable of decoding and recovering the encoded image file back to original by a predetermined corresponding decoding method.
In accordance with psychology of vision, an individual edge would not affect visional recognition. However, such an individual edge may be mutually affected with edges adjacent thereto so as to generate a composition effect. In this concern, human vision is more sensitive to edge distortions. Generally, edge distortions include blurred edges and newly created false edges. As such, it is very important to maintain original edges and avoid the creation of false edges when executing an image process.
A method of recovering or obtaining image content of a damaged MB according to image content of correct MBs of a same frame of the damaged MB is generally called the SEC method.
There are many algorithms of conventional technologies have been proposed for SEC, such as bilinear interpolation (BI), directional interpolation (DI), and best neighborhood matching (BNM). The principles of the aforementioned algorithms can be learnt in more details by referring to related references, such as: P. Salama, N. B. Shroff, and E. J. Delp, “Error concealment in encoded video streams,” in Signal Recovery Techniques for Image and Video Compression and Transmission, A. K. Katsaggelos and N. P. Galatsanos, Eds. Norwell, M A: Kluwer, ch. 7, 1998; H. Sun and W. Kwok, “Concealment of damaged block transform coded images using projection onto convex set,” IEEE Trans. Image Processing, vol. 4, pp. 470-477, April 1995; and Z. Wang, Y. Yu, and D. Zhang, “Best neighborhood matching: An information loss restoration technique for block-based image coding systems,” IEEE Trans. Image Process., vol. 7, no. 7, pp. 1056-1061, July 1998. They are to be briefly illustrated herebelow for better exemplification of the present invention.
FIG. 1 is a schematic diagram illustrating a BI method according to a conventional technology. Referring to FIG. 1, a damaged MB 20 having no correct image content is surrounded by edge reference pixels 21 containing correct image content. A missing pixel P(x, y) in the damaged MB 20 positioned at the coordinates (x, y) of BI is then interpolated by the formula (1), according to the correct image content of the four edges in both horizontal direction and vertical direction:
                                          p            ⁡                          (                              x                ,                y                            )                                =                                                    p                ⁢                                                                  ⁢                1                ×                d                ⁢                                                                  ⁢                2                            +                              p                ⁢                                                                  ⁢                2                ×                d                ⁢                                                                  ⁢                1                            +                              p                ⁢                                                                  ⁢                3                ×                d                ⁢                                                                  ⁢                4                            +                              p                ⁢                                                                  ⁢                4                ×                d                ⁢                                                                  ⁢                3                                                                    d                ⁢                                                                  ⁢                1                            +                              d                ⁢                                                                  ⁢                2                            +                              d                ⁢                                                                  ⁢                3                            +                              d                ⁢                                                                  ⁢                4                                                    ,                            (        1        )            in which d1, d2, d3, d4 are relative distances from the edge pixels P1, P2, P3, P4 to the missing pixel P(x, y), respectively.
FIG. 2 is a schematic diagram illustrating a DI method according to a conventional technology. As shown in FIG. 2, the edge pixels contain correct image content, while the MB surrounded by the edge pixels is a damaged MB containing incorrect image content. The pixel P(x, y) in the damaged MB positioned at the coordinates (x, y) is then interpolated by the formula (2), according to the correct image content of two edges along a specific direction:
                                          p            ⁡                          (                              x                ,                y                            )                                =                                                    p                ⁢                                                                  ⁢                1                ×                d                ⁢                                                                  ⁢                1                            +                              p                ⁢                                                                  ⁢                2                ×                d                ⁢                                                                  ⁢                2                                                                    d                ⁢                                                                  ⁢                1                            +                              d                ⁢                                                                  ⁢                2                                                    ,                            (        2        )            in which d1, d2 are relative distances from the edge pixels P1, P2, to the missing pixel P(x, y), respectively.
FIG. 3 is a directional schematic diagram illustrating a DI method according to a conventional technology. As shown in FIG. 3, in the DI method, the edge direction is classified into 8 directions, in which DI(0°) represents a direction of 0°, DI(22.5°) represents a direction of 22.5°, . . . and so forth. P1 and P2 of FIG. 2 are each of one of the 8 directions.
DI method usually employs a direction filter of an edge direction detection technique for determining direction of each pixel contained in the MBs surrounding the damaged MB. Sobel operator or Prewitt operator are often used. Sobel operator, which is also known as a Sobel filter, is to be exemplified for illustration below taking a 3×3 Sobel operator as an example:
                              S          x                =                                            [                                                                                          -                      1                                                                            0                                                        1                                                                                                              -                      2                                                                            0                                                        2                                                                                                              -                      1                                                                            0                                                        1                                                              ]                        ⁢                          S              y                                =                      [                                                                                -                    1                                                                                        -                    2                                                                                        -                    1                                                                                                0                                                  0                                                  0                                                                              1                                                  2                                                  1                                                      ]                                              (        3        )            in which Sx and Sy detect edge characteristics at X direction and Y direction, respectively, and obtain directional gradients at X direction and Y direction of equation (4) (Gx and Gy), respectively.GX=Xi+1, j−1−Xi−1, j−1+2Xi+1, j−2Xi−1,j+Xi+1,j+1−Xi−1,j+1 Gy=Xi−1, j+1−Xi−1, j−1+2Xi, j+1−2Xi,j−1+Xi+1,j+1−Xi−1,j−1   (4).
A directional gradient (Grand) and a directional angle (θ) can be further calculated with equations (5) and (6).Grand=√{square root over (Gx2+Gy2)}  (5)θ=tan−1(Gx/Gy)   (6)in which the directional gradient (Grand) is a quantity factor for evaluating the strength of the directivity. In other words, a directional angle (θ) having a maximum directional gradient (Grand) is selected serving as a direction reference for the damaged MB.
If all of the directional gradients (Grand) are too low, or even lower than a threshold, the damaged MB is determined as having no directivity, and not adapted for SEC by DI method. As such, other methods, (e.g., BI method), are demanded for processing those damaged MBs having no directivity.
FIG. 4 is a schematic diagram illustrating a BNM method according to a conventional technology. Referring to FIG. 4, a damaged MB 20 of frame 40 is surrounded by a plurality of reference MBs 22, and a best matching MB 32 is surrounded by a plurality of target MBs 30 in a searching area 42. The BNM method is for finding out the best matching MB 32 for replacing the damaged MB 20. The BNM method includes the steps of: (1) taking N-pixel-wide boundary surrounding the damaged MB as a searching image; (2) identifying best matching MBs nearest to the damaged MB; and (3) replacing the corresponding damaged MB (i.e., the MB surrounded by the best matching MBs) with the identified best matching MB.
Features of the BI method, ID method, and BNM method are to be discussed below in further details.
The DI method has the advantages of edge protection. For example, if there is only one strong edge near the damaged MB, the original strong edge can be preserved and avoided from becoming blur. However, when there are several edges near the damaged MB, the DI method may generate several false edges. Unfortunately, human vision is usually very sensitive to blur false edges. Correspondingly, the BI method is adapted to generate new blur false edges. However, if there is only one strong edge near the damaged MB, the BI method may blur the original strong edge because the edges of the damaged MB are interpolation processed and mixed with the adjacent pixels. Further, with respect to damaged MBs having patterns of higher complexity, it is often difficult to obtain necessary effective reference blocks. Although the BNM method can recover the complex damaged MB, it may cause edge discontinuity. It may also cause higher computation complexity than BI and DI methods, and thus is featured with a lower overall efficiency.
As such, an efficient SEC method which is adapted for preserving original edges while avoiding generation of new false edges, and can be conducted with less calculation amount, is desired.